It’s not often that a zero-day vulnerability causes a network security vendor to urge customers to physically remove and decommission an entire line of affected hardware — as opposed to just applying software updates. But experts say that is exactly what transpired this week with Barracuda Networks, as the company struggled to combat a sprawling malware threat which appears to have undermined its email security appliances in such a fundamental way that they can no longer be safely updated with software fixes.
The Barracuda Email Security Gateway (ESG) 900 appliance.
Campbell, Calif. based Barracuda said it hired incident response firm Mandiant on May 18 after receiving reports about unusual traffic originating from its Email Security Gateway (ESG) devices, which are designed to sit at the edge of an organization’s network and scan all incoming and outgoing email for malware.
On May 19, Barracuda identified that the malicious traffic was taking advantage of a previously unknown vulnerability in its ESG appliances, and on May 20 the company pushed a patch for the flaw to all affected appliances (CVE-2023-2868).
In its security advisory, Barracuda said the vulnerability existed in the Barracuda software component responsible for screening attachments for malware. More alarmingly, the company said it appears attackers first started exploiting the flaw in October 2022.
But on June 6, Barracuda suddenly began urging its ESG customers to wholesale rip out and replace — not patch — affected appliances.
“Impacted ESG appliances must be immediately replaced regardless of patch version level,” the company’s advisory warned. “Barracuda’s recommendation at this time is full replacement of the impacted ESG.”
In a statement, Barracuda said it will be providing the replacement product to impacted customers at no cost, and that not all ESG appliances were compromised.
“No other Barracuda product, including our SaaS email solutions, were impacted by this vulnerability,” the company said. “If an ESG appliance is displaying a notification in the User Interface, the ESG appliance had indicators of compromise. If no notification is displayed, we have no reason to believe that the appliance has been compromised at this time.”
Nevertheless, the statement says that “out of an abundance of caution and in furtherance of our containment strategy, we recommend impacted customers replace their compromised appliance.”
“As of June 8, 2023, approximately 5% of active ESG appliances worldwide have shown any evidence of known indicators of compromise due to the vulnerability,” the statement continues. “Despite deployment of additional patches based on known IOCs, we continue to see evidence of ongoing malware activity on a subset of the compromised appliances. Therefore, we would like customers to replace any compromised appliance with a new unaffected device.”
Rapid7‘s Caitlin Condon called this remarkable turn of events “fairly stunning,” and said there appear to be roughly 11,000 vulnerable ESG devices still connected to the Internet worldwide.
“The pivot from patch to total replacement of affected devices is fairly stunning and implies the malware the threat actors deployed somehow achieves persistence at a low enough level that even wiping the device wouldn’t eradicate attacker access,” Condon wrote.
Barracuda said the malware was identified on a subset of appliances that allowed the attackers persistent backdoor access to the devices, and that evidence of data exfiltration was identified on some systems.
Rapid7 said it has seen no evidence that attackers are using the flaw to move laterally within victim networks. But that may be small consolation for Barracuda customers now coming to terms with the notion that foreign cyberspies probably have been hoovering up all their email for months.
Nicholas Weaver, a researcher at University of California, Berkeley’s International Computer Science Institute (ICSI), said it is likely that the malware was able to corrupt the underlying firmware that powers the ESG devices in some irreparable way.
“One of the goals of malware is to be hard to remove, and this suggests the malware compromised the firmware itself to make it really hard to remove and really stealthy,” Weaver said. “That’s not a ransomware actor, that’s a state actor. Why? Because a ransomware actor doesn’t care about that level of access. They don’t need it. If they’re going for data extortion, it’s more like a smash-and-grab. If they’re going for data ransoming, they’re encrypting the data itself — not the machines.”
In addition to replacing devices, Barracuda says ESG customers should also rotate any credentials connected to the appliance(s), and check for signs of compromise dating back to at least October 2022 using the network and endpoint indicators the company has released publicly.
Update, June 9, 11:55 a.m. ET: Barracuda has issued an updated statement about the incident, portions of which are now excerpted above.
Who will pay for the replacement hardware? I did not see in the announcement that Barracuda would replace the hardware at no charge to the customer, or that they would reimburse the labor costs required.
Clearly, these devices are likely not “merchantable or fit for its intended purchase.” Legally that could mean the manufacturer will be required to replace them without cost to the buyer regardless of warranty limits and exculpatory language in their various agreements. Consequential losses, such as the labor to replace units, are legally easier for the manufacturer to avoid, and they will likely be a matter of some negotiation, regulatory action, or litigation to determine who will pay these costs. The manufacturer may claim a defense of force majeure, a legal term for an act of god or an act of war, but that would be a reach if the attack could have been reasonably foreseen (or we’re not soon at war with the state actor initiating the attack). Anyway you look at it; it’s an existential crisis for the manufacturer.
I don’t think, that this leads to an “existential crisis for the manufacturer”. A lot of customers will pay for an “Instant Replacement Subscription”, and for this service Barracuda charges half the price of the new appliance per year. So every customer, who has such an subscription since 2022 has fully paid – in advance – for the replacement.
And further I don’t think, that Barracuda makes much money with the hardware – customers have to pay annual user licences for each user, which very fast will amount to much higher returns than from the hardware. And the licences are in no way affected from this security desaster.
Can we just stop with this mythical thing called “not merchantable or fit for its intended purpose” nonsense as I’m more than sure that Barracuda as well as any other hardware/software manufacturers/producers have specific disclaimers stating that they do not guarantee merchantability and pass the determination of “fitness for intended purpose” up to the buyer to decide.
So no matter how you might look at it there isn’t any “existential crisis” for the manufacturer.
Nope. No existential crisis for the manufacturer because the Terms of Service/Use expressly disclaim implied warranty of merchantability and fitness for a particular purpose. Schedule #2: Limited Warranty.
Source:
https: //www.barracuda.com/company/legal/terms-and-conditions/limited-warranty
It’s important that they get the pertinent point out that “impacted” devices are to be considered
untrusted regardless of patch level after the fact, whether or not they’re landfill or refurbishable.
If they don’t offer some sort of RMA replacement there would be a huge stink so I’d expect that.
Thank you for getting to the heart of the matter and putting into precise language what must have been in most readers’ minds by the second paragraph!
“In a statement, Barracuda said it will be providing the replacement product to impacted customers at no cost, and that not all ESG appliances were compromised.”
Reading is fundamental.
“In a statement, Barracuda said it will be providing the replacement product to impacted customers at no cost, and that not all ESG appliances were compromised.”
If you read the article, you’ll see: “In a statement, Barracuda said it will be providing the replacement product to impacted customers at no cost, and that not all ESG appliances were compromised.”
Any manufacturer of “security equipment” that fails this comprehensibly SHOULD be in existential crisis.
If we allow the “Corporations are people too” argument to stand (it shouldn’t), this person should receive the death penalty.
Thanks, Brian, and Krebs. This is good as a start. Looking forward to hearing the full nature of why the hardware should be tossed. While I sympathize with the people above, why is morr important than “who will line my pockets.”
Krebs does all the work. Brian is just the face of the operation.
Quite obviously, because the hardware is too compromised to be salvageable. I can’t think of any other scenario where they ‘d do this.
Not necessarily, but at the moment don’t trust end user patches to fully remove existing compromise.
Patching only (maybe) halts a vector of initial compromise, not detect/remove subsequent payloads.
It doesn’t mean hardware is “unsalvageable,” it means the process for salvaging is more intensive
(or at this point unknown) so they can’t recommend action other than unit replacement at this time.
The advice is important to prevent users from patching over a deeper compromise unknowingly,
assuming the device is now secure on that basis. They have to research a secure restoration process.
It may be that it’s simpler for the company to eat the cost of replacement with new units instead.
We’ll see.
Can’t help but wonder if the equipment *could* be salvaged but Barracuda has decided this is the perfect opportunity to force people to their cloud SaaS solution.
Sorry, but “Email Security Gateway” sending out malware sounds funny.
The question is, who would trust such a vendor in the future?
Is there a single vendor that hasn’t had a major failure? They are taking the right steps to keep their clients safe. How many would just pretend or do some kind of promotion to encourage replacement and create some BS spin? They are tackling it head on and after more than 3 decades in IT, I respect that.
There are unpatchable hardware problems, sometimes. But if the manufacturer tells me to replace hardware I always wonder if the problem really is hardware being unsafe or if the manufacturer doesn’t want to support the old boxes any more/wants me to buy new ones anyway.
Why are you choosing to be a paranoid person?
Guess the boxes can’t be customer booted from USB, or possible that they have UEFI bugs that have been left unpatched, so not even the bootloader can be trusted. Maybe there are evidence of hacked UEFI firmware. If we are there, then only hardware replacement can fix the issues, unless you replace the ROM, which is most likley soldered.
I have been playing with different microelectronics, and on some the bootloader becomes corrupted, but then there usually is some way to pull pins high/low that will put a microprocessor into flash mode. But this might be impossible when everything is soldered.
Its not clear anywhere I search: Are these Barracuda models or software that is past its end-of-life? Or is Barracuda pressing for removal of current and supported models of their technology?
Either way, not great. But I have less sympathy for the customer that was using obsolete and no longer supported technology if that’s the case.
If its supported technology, should be able to RMA the device and get a newer unaffected model in return from the manufacturer. Until they run out anyway..
Barracuda said the vulnerability affects a subset of devices running versions 5.1.3.001 – 9.2.0.006. That seems like a pretty broad range.
This is a bold move by a company. Most would hide the issue because of the fear this would tarnish the brand. I love the honesty and integrity of the brand that admits they can’t fix something and does what is right for their customers. Much respect!
Yes, Steve, I agree.
I am reminded of the 1982 Tylenol murders and the prompt, effective and open response by Johnson & Johnson.
The murderers had used cyanide to contaminate a legitimate product. They were never caught. Tylenol is still a best-selling pain relief medicine.
https://www.pbs.org/newshour/health/tylenol-murders-1982
Barracuda did not intentionally manufacture and sell defective devices. We may never know who created the malware.
They might not have done it intentionally, but this is unforgiveable.
“The command injection vulnerability exists in the parsing logic for the processing of TAR files. The following code within the product is the focal point of the vulnerability:
qx{$tarexec -O -xf $tempdir/parts/$part ‘$f’};
It effectively amounts to unsanitized and unfiltered user-controlled input via the $f variable being executed as a system command through Perl’s qx{} routine. $f is a user-controlled variable that will contain the filenames of the archived files within a TAR.”
“but this is unforgiveable.”
Implying some majority of everything else is tacitly forgivable.
Gee, I wonder how many of the corps that own these 11,000 devices are going to be replacing their Barracuda ESGs with ones from the same manufacturer?
Personally, I’d be looking for a new brand.
If the company offers 1:1 replacement I’d imagine most would take them up on that.
Any indications the malware was introduced in the supply chain?
Who will bear the cost of replacing the devices? Would this considered part of the support contract as an uncorrectable defect?
The victimized businesses probably have insurance claims with their own insurance companies under multiple lines of coverage. Document all replacement activities and costs, including delays, labor expenses, business interruption etc. Of course, any deductibles and Self Insurance Retentions apply.
Be aware that Cyber Insurance Policies frequently have multiple sublimits and special deductibles for each category of loss. Have someone very knowledgeable about insurance coverage, especially manuscript Cyber Insurance coverages, review your policies immediately. If your insurance pays the claims, the insurance carrier will be subrogated and also able to assert claims to recover your deductibles and SIRs as well as carriers own payments against Barracuda.
Likely they will not replace hardware and push customers to the SaaS solution.
There were 2 cases similar to this in the past, one with Cisco ESA and the other one with Symantec Email Gateway. I know some people that preferred to replace their appliances instead of patching, however both vendors confirmed the patch was effective. I only know 3 other cases where the hardware was replaced, the Meltdown problem from Intel, NVidia with Spectre and Apple on some laptops because of a problem with bootcamp. Was ever something like this on a security appliance?
Firmware compromise is nothing new. The NSA was using this tactic against HDD manufacturer bios/firmware. Basically they used the HDD firmware for persistence and could execute against several HDD major manufacturers.
My isp is most likely a victim of this issue. I have service and an email address through Windstream. I received an email with my password. Clients of the business I work for received emails with their passwords as well.
Many of them suffered password resets of their social media and other accounts. Windstream has yet to own up to being compromised.
Windstream is using FireEye Cloud. Unless they changed it recently.
Re: Bob Collins comments.
That depends on the actual policy terms and provisions. As always with insurance, RTFP! Translation from lawyerese, Read the Fine Print or Read the Full Policy . . .
Most Cyber Insurance policies will replace some, perhaps all of your hardware and more. Last I reviewed the topic for a paper I am writing, there were over 400 separate cyber coverage policies and provisions in the market. If the business does not have Cyber Insurance, there may still be so called “Silent Cyber” potential coverage through some Crime, Professional, CGL etc. These are often “sub-limited” to absurdly insufficient amounts, but there may be at least some coverage.
In any event, every affected business needs to:
1. actually obtain their full policies, NOT merely the Declarations Page! Dec pages can be horribly misleading using terms like “All Risk” though the policy definitely does NOT cover all risks. Think of the obscure, nearly unknown, “Virus Exclusion” to Business Insurance.
2. assemble ALL your insurance policies of every kind and purpose and carefully read them. Again, RTFP! If you do not understand every clause, definition, word, and phrase obtain help from a coverage attorney in your state(s). Insurance is regulated at the state level in the U.S. and can be differently interpreted in the various states.
And, IF you are going to try to read the policies yourself, . . . some adult libation may ease the boredom and reduce the eye glazing. 😉
This illustrates the fundamental griftiness of “security” vendors.
Who can you trust? Someone who once sold you something? That seems fairly foolish, doesn’t it? Customers are sunk cost, assuming a target-rich environment.
Outsourcing is to voluntarily impoverish your internal expertise. It’s not just a cost-cutting tweak, though you may in fact be comfortable to impoverish your, for instance, grounds-keeping or catering resources. But can you afford to voluntarily rid yourself of security expertise? Maybe you can! It’s really a question of how well the outsourced product works, what it costs, including its risk and impact of failure. The problem is that a security failure can be catastrophic, unfixable.
Worse than catastrophic or “unfixable” is unnoticed, unknown, silently undermined ongoing.
Do you really believe your own security expertise is greater than what any/all vendors offer?
When “own security” kicks it you won’t likely get Mandiant warning you of unusual traffic.
Are we experts of every system we’d use in a given day, claim that with a straight face?
(Ken Thompson’s still alive…) Even hand-doping your own Billions of logic gates 1 by 1,
you rely on many (!) things you can neither create from scratch nor realistically audit solo.
Time is finite. Expertise is a work product you don’t have time on Earth to get all yourself,
& possible future failure conditions are open-ended and infinite. Trusting is inevitable; thus
trust being violated is on that level inevitable. Weighing risk is compromise in estimates,
& superlatives have no actual value beyond the vague misconceptions they instill in others:
There’s no such thing as “secure,” there are test cases with various assurances tacked on.
Those seeking “total security” may as well seek a priest or shrink, skip straight to a druggist.
This means China has 8 months of plaintext ingress emails and attachments from targeted enterprises because of their email security system. Despite Rapid7’s seeming assurance that they did not see lateral movement, there is no way that a nation state actor using such a specific zero day did not attempt to identify pivot opportunities. Since they had access to plain text email, some really nice spear fishing was also likely performed. For instance, they had the ability to extend a thread with an updated attachment that contained a payload. These organizations now have a lot more to worry about than the spilled milk. Their first line defenses have been unknowingly breached for months.
So, this is why we need to start encrypting all our emails (internal and external). Unfortunately, we are so multi-homed (laptops, desktops, phones, even TV’s and doorbells) it makes key management very difficult to roll out to the masses.